Compositional Sentence Representation from Character Within Large Context Text

In this work, we targeted two problems of representing a sentence on the basis of a constituent word sequence: a data-sparsity problem in non-compositional word embedding, and no usage of inter-sentence dependency. To improve these two problems, we propose a Hierarchical Composition Recurrent Network (HCRN), which consists of a hierarchy with 3 levels of compositional models: character, word and sentence. In HCRN, word representations are built from characters, thus resolving the data-sparsity problem. Moreover, an inter-sentence dependency is embedded into the sentence representation at the level of sentence composition. In order to alleviate optimization difficulty of end-to-end learning for the HCRN, we adopt a hierarchy-wise learning scheme. The HCRN was evaluated on a dialogue act classification task quantitatively and qualitatively. Especially, sentence representations with an inter-sentence dependency significantly improved the performance by capturing both implicit and explicit semantics of sentence. In classifying dialogue act on the SWBD-DAMSL database, our HCRN achieved state-of-the-art performance with a test error rate of 22.7%.